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aimed  at  improving  the  test  and  evaluation  of  major  systems  through  improved 
acquisition  management  and  risk  reduction  procedures.  This  manual  contains 
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Evaluation. 
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Washington,  D.C.  20301-3110 
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Volwng  I 

Guidelines  for  the  Trea'H»i«=*^t  Software  in 
Test  and  Evaluation  Master  Plans 

Section  1 
Notes  to  the  Oser 


This  manual  is  intended  to  support  the  review  of  Test  and  Evaluation 
Master  Plans  (TEMPs)  for  systems  that: 

1.  contain  mission  critical  software  conponentSf 

2.  are  software  intaisiver  or 

3.  presait  software  testing  issues  that  significantly  affect  ri^. 


The  primary  audiaice  for  this  manual  consists  of  those  individuals  who 
are  responsible  for  evaluating  TEMPs,  However,  this  manual  should  ^Iso  prove 
useful  to  program  offices,  indepaident  test  organizations,  contractors,  and 
others  who  review,  prepare,  or  provide  data  for  inclusion  in  TQlPs, 

Tie  overall  organization  of  Section  2  of  this  volume  is  keyed  to  the  TEMP 
structure  defined  by  DoD  Directive  5000.3,  "Test  and  Evaluation"  (December, 
1979;  Enclosure  2) ,  It  should  be  possible  to  evaluate  a  TEMP  using  this 
portion  of  the  docum^t  as  a  roadnap  of  the  software  issues  that  may  arise. 
Section  2  contains: 

Tie  TEMP  and  Software  Checklist:  The  Checklist  is  a  series  of 
questions  keyed  to  the  major  paragra^is  and  sections  of  a  TEMP.  In 
format,  the  questions  are  phrased  to  permit  sinple  "yes"  or  "no" 
answers.  In  practice,  it  is  not  necessary  that  every  checklist 
question  be  answered  by  the  TEMP,  That  is,  a  checklist  question  may  be 
inapplicable  to  the  system  being  reviewed.  In  other  cases,  checklist 
questions  that  are  not  answered  by  the  TEMP  may  indicate  deficiencies. 

Explanatory  Notes:  The  Motes  are  brief  onmnentaries  on  the  ohecklist 
questions  and  the  significance  of  the  possible  responses  to  them.  The 
manual  user  needs  to  be  aware  that  the  Notes  are  of  a  general  nature 
and  may  not  aoxiurately  refleort:  the  intricacies  of  the  teohnology  used 
in  a  given  system.  Nevertheless,  the  Notes  describe  issues  that  a 
software  aigineer  would  raise  in  evaluating  whether  or  no)t  the  software 
has  demonstrated,  through  extensive  usage,  testing,  and  r^ir,  that  it 
meets  design  and  user  requiremoits.  Therefore,  these  issues  should  be 
of  primary  concern  when  evaluating  a  TEMP, 

Each  of  the  checklist  questions  references  an  &q>lanatory  note.  The  notes 
^jpear  on  the  page  opposite  the  question  that  references  them.  This  logout 
has  been  chosen  with  the  goal  of  minimizing  the  amount  of  "paging"  required 
vhen  using  this  manual. 


Section  3  consists  of  the  References  and  Glossary,  Hie  Checklist  and  Notes 
are  not  intended  to  be  a  "textbodc"  in  software  testing.  Variations  in 
terminology  and  basic  software  definitions  are  accounted  for  in  the  Glossary, 
Detailed  discussions  of  technical  matters  that  are  mentioned  in  the  Notes  are 
contained  in  the  References, 
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•3 


^gJSection/Subsection 


Checklist  Ouestic»is 


^rt  I  -  Description 

1.  Mission,  niis  section  sumnarizes 
operational  need,  mission  to  be 
acconplisQied  and  planned  opera¬ 
tional  environment  and  relates 
direo±ly  to  the  Mission  Element 
Need  Statement  (MENS)  and  planned 
^stem  operational  concept. 


(Note  0) 
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fbte  Qt 


The  description  of  the  missiai  to  be  performed  should  be  a  statement  of  need 
and  operational  conc^t.  Software  specific  referoices  are  generally  not 
appropriate  in  this  section.  A  possible  e}K:epticai  may  occur  if  the  only  way 
to  egress  the  system  operational  concept  is  in  software-oriented 
terminology. 

References  for  Note  0;  [Redwine,  Chapter  II] 
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;gMP  SectiCTi/Subsection 


Checklist  CXiestiois 


I^rt  I  -  Description 

2.  System.  This  section  is  a  brief 
description  of  the  system 
including  discussicais  of  key 
functions,  interfaces,  and 
unique  characteristics. 


Does  the  system  contain  mission 
critical  conpjter  resources? 
Oiote  1) 


ms.  1; 


The  term  "Mission  Critical  Computer  Resources  (MCCR) "  is  defined  by  Section 
908  of  the  EY  1982  Defense  Authorization  Act,  ICCR  include  automatic  data 
processing  equipment  or  services  whose  functions  are  critical  to  si:^^rt: 

—  intelligaice  activities 

—  cryptologic  activities  related  to  natiaial  security 

—  conmand  and  control  of  military  forces 

—  eguipnent  that  is  an  integral  part  of  a  weapon  or  weapon  system 

—  the  direct  fulfillmait  of  military  or  intelligence  missicxis. 

Conputers  for  MCCR  applications  are  exempted  frcan  the  provisions  of  the 
Brooks  Act  (P.L,  89-306) ,  Acquisition  policy  for  MCCR  is  defined  by  DoD 
Directive  5000.29  ("Management  of  Computer  Resources  in  Major  Defaise 
Systems") . 

Prom  a  technical  point  of  view,  knowing  vdiether  or  not  the  application 
contains  MCCR  is  often  critical  in  determining  the  role  that  software  plays 
in  ccxit rolling  systems  functions.  Therefore,  it  is  essential  that  the  TEMP 
be  sufficiently  detailed  to  answer  this  quest icxi. 

If  the  system  contains  an  MCCR  applicatiai,  the  TEMP  should  allow  an  easy 
determination  of  how  critical  softvare  is  to  the  essential  missiai  functions. 
One  possibility  is  that  mission  objectives  are  achieved  by  special-purpose 
conputers  or  circuitry  in  which  software  instructions  play  no  important  role. 
These  situations  are  rare,  since  a  key  role  of  MCCR  technology  is  to  provide 
for  systems  that  are  easily  changeable  (e.g.,  to  adapt  to  changing  threats). 
Another  possibility  is  that  the  system  contains  an  MCX2R  application  and  that 
software  plays  a  significant  role. 

Hie  kind  of  MCCR  application  may  point  to  areas  in  which  special  T&E 
considerations  should  be  takai  into  account.  For  example,  there  may  be 
secure  functions  whose  implanentations  should  be  certified  by  the  DoD 
Security  Caiter, 

Hie  nature  of  an  MCCR  application  may  also  lead  to  managemait  issues  that 
affect  risk.  For  example,  the  use  of  MCXIEts  in  some  applications  leads  to 
specialized  Service  standards  and  regulations;  these  may  by  themiselves  give 
rise  to  T&E  issues.  The  same  considerations  may  also  affect  inter-agency 
agreements  for  scheduling  and  budgeting  and  down-stream  problems  such  as  test 
data-sharing  and  test-bed  availability. 

Even  if  the  ^KXIR  aj^licaticxi  involves  neither  software  intensive  applications 
(cf.  Mote  2)  nor  missicxi  critical  software  (cf.  Note  3),  there  may  be 
significant  software  T&E  issues.  For  example,  in  many  real-time  systems,  a 
degree  of  fault-tolerance  can  be  gained  software  (e.g.,  providing  for 
graceful  degradaticxi  of  communicating  processors) .  System  designers  may  have 
ignored  such  opportunities  and,  as  a  result,  the  system  may  have  few  software 
features  that  protect  users  from  faulty  hardware.  In  such  an  instance,  the 
l&gK  of  software  features  should  be  treated  as  a  design  flaw  that  will 
ultimately  reduce  system  availability. 

Referoices  for  Note  1:  [Grovel ,  [Redwine,  Chapter  III,  [STEPll, 

[STEPS,  Nelson],  [STEPS,  Stewart! 
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Checklist  Ouesticais 


Part  I  -  Description 

2,  S^stsn.  This  section  is  a  brief  _  Is  the  system  software  intensive? 

description  of  the  system  CNote  2) 

including  discussions  of  key 
functions,  interfaces,  and 
unique  characteristics. 


tfcte  2; 


If  the  system  does  not  contain  M(XR,  the  software  may  still  contribute  to 
overall  risk,  (Non-41CX3l  applications  are  generally  not  subject  to  the 
policies  governing  MOCR  systems;  see  Note  1,)  This  is  often  the  case  vAien 
the  bulk  of  system  development  is  devoted  to  software,  to  exanple  of  such  a 
system  might  be  an  automatic  data  processing  (ADP)  application  in  v^ich  the 
system  hardware  is  an  off-the-toelf  conmercial  product,  but  the  softvrare 
needs  to  be  developed  to  meet  military  requirements.  In  these  situations, 
the  operational  characteristics  of  the  commercial  hardware  cannot  be 
transferred  to  the  software  (e.g.,  ease  of  maintenance  for  the  hardware  does 
not  alleviate  poteitial  maintenance  problems  for  the  software) . 

Other  sources  of  risk  arise  vrtien  the  software  is  likely  to  be  difficult  to 
design  or  requires  the  use  of  new  or  undemonstrated  software  engineering 
practices  and  tools. 

Even  if  the  system  does  not  contain  mission  critical  computer  resources,  it 
is  software  intensive  if  it  contains  software  that: 

—  dominates  the  develcpnent  budget 

—  is  large  or  conplex  relative  to  overall  system  size  and  complexity. 

It  is  usually  prudent  to  treat  software  intensive  systems  as  if  they 
contained  mission  critical  computers  and  software.  For  the  T&E  community, 
these  systems  pose  special  problems,  The  framers  of  the  system  development 
plans  and  the  TEMP  are  responsible  for  ensuring  that  proper  consideration  is 
givQi  to  the  testing  of  the  software  in  these  ^sterns. 

Qie  aspect  of  the  TEMP  that  needs  special  attention  for  software  intensive 
systems  —  especially  for  ncxi-MOCR  systems  —  is  the  description  of 
ftinctional  c^>abilities  that  will  be  demonstrated  by  testing.  For  non-MOCR 
i^stems,  these  may  be  related  only  remotely  to  operational  missioxi 
objeo±ives.  Therefore,  these  o»pabilities  are  easy  to  overloxrfo  at  the  system 
level. 

A  o»reful  review  of  required  system  characteristics  and  critical  interfaces 
(see  Notes  5  and  7  below)  to  determine  thoxse  that  are  traceable  to  software 
requirements  may  be  helpful.  Functional  areas  in  which  non-MCXK  software 
plays  an  important  role  include  the  following: 

—  data  base  management 

—  cornmunicatioxi  and  networking 

—  CAD/CAM  and  support  software  development 

—  training  and  simulation 

—  oxmiputer  graphics  and  humein  interfaces 

—  decision  support, 

A  system  having  functional  c^>abilities  in  any  of  these  areas  —  or  having 
signifio:ant  interfaces  with  systems  that  provide  these  capabilities  ■—  is 
prohiably  software  intensive.  The  test  and  evaluation  of  the  software  in 
these  systems  should  be  planned  and  managed  as  if  the  system  contained 
mission  critical  software, 

Referaices  for  Note  2:  [Bunyardl,  [SPEPl] 
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Checklist  Quest ic»is 


I  -  Degcripticn 

2,  This  section  is  a  brief  _  Does  the  software  ixnplement 

description  of  the  system  critical  functions?  (Note  3) 

including  discussions  of  key 
functions,  interfaces,  and 
unique  characteristics. 
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Note  3t 


An  adequate  risk  assessment  requires  that  this  question  be  answered  at  the 
system  level.  If  the  system  contains  mission  critical  computer  resources 
(see  Note  1)  or  if  it  is  software  intensive  (see  Note  2),  thoi  the 
capabilities  to  be  performed  by  the  software  should  be  e3q>licitly  cited. 
Failure  to  highlight  software-iirplanaited  mission  critical  functions  may  lead 
to  tests  that  do  not  adequately  assess  ccmfonnance  to  technical  and 
operational  objectives. 

In  addition,  the  lack,  of  software-inplanaited  functicxis  may  in  some 
circumstances  be  questioned.  For  example,  a  system  that  is  supposed  to  be 
capable  of  rapid  adaptation  but  has  only  hardware  implanentaticxi  of  its 
critical  functions  may  exhibit  unacceptable  availability  characteristics  in 
operation. 

Hie  paramount  issue  that  arises  from  a  "YES"  answer  to  this  questicxi  is 
v^ether  the  software  has  beai  given  balanced  treatment  with  other  critical 
^stem  coinpaiQits.  An  early  determinatiai  of  the  software-inplemented 
functicxis  allows  for  careful  design  of  testable  software  requirements  and 
specifications  and  the  development  of  a  systematic  approach  to  software 
testing.  Experience  has  shown  that  vrtioi  these  considerations  are  pushed 
later  into  the  development/acquisition  process,  latent  problems  with  the 
software  are  more  difficult  to  eliminate  and  the  resulting  systems  are  less 
well-suited  to  their  objectives. 

To  emphasize  the  balance  that  should  be  sought  betweai  the  software  and  the 
hardware  componaits  that  implement  critical  functions,  the  term  critical 
software  componait  will  be  used  throughout  this  manual.  A  critical  software 
component  is  any  portioi  of  a  computer  program,  any  computer  program,  or  any 
collection  of  computer  programs  that  fulfills  a  requiremoit  for  a  Key 
Function  as  described  in  the  TEMP. 

References  for  Note  3t  [Redwine,  Chapter  III],  [STEPl],  [STEPS,  Part  2] 


-11- 


2.  Siystem 


a.  Kgy  Fyinctions:  these  are 
functiois  that  enable  the 
^stem  to  accOTplish  its 
operational  mission;  this 
description  may  include  a 
mission/function  matrix 
relating  primary  functional 
capabilities  that  must  be 
demonstrated  by  testing  to 
the  mission  to  be  performed 
and  the  concept  of  operatiai. 


_  Does  the  Missiai/Function  matrix 

identify  primary  functional 
capabilities  to  be  iitplemented  by 
the  software?  (Note  4) 

_  Are  the  identified  functions 

implemaited  in  software: 

_  New?  (Note  4a) 

_  Autonation  or  modification 

of  existing  capabilities? 
(Note  4>) 

_  Mature?  (Note  4c) 
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Note  4! 


The  miss  ion/ function  matrix  (or  equivalent  narrative)  is  the  primary  source 
of  information  about  how  the  capabilities  have  been  partitioned  between 
hardware  and  software.  These  partitions  will  be  important  in  determining 
required  characteristics,  in  defining  error/ failure  categories,  and  in 
isolating  and  correcting  deficiencies  noted  during  testing.  Therefore,  it 
may  be  important  to  determine  that  proper  engineering  studies  have  led  to  the 
establishment  of  these  partitions. 

An  understanding  of  the  sources  of  risk  in  each  of  the  software-ijrplero«ited 
functions  idoitified  in  the  miss  ion/ function  matrix  is  an  ess^tial  part  of 
the  overall  risk  assessmoit.  Some  typical  risk  drivers  are  those  vhich 
influence  the  maturity  of  the  software. 

a.  New  Function?  New  software  functions  generally  r^resent  the  highest 
risk,  since  they  involve  not  only  the  design  of  software,  but  also 
the  use  of  new  concepts,  theories  and  algorithms.  These  functions 
are  oftai  found  in  applicatiais  of  emerging  critical  technologies 
such  as  artificial  intelligence,  distributed  data  processing,  or 
ultra-reliable  ccirputing.  Often,  these  functions  have  only  been 
demonstrated  in  the  laboratory  and  no  operator  personnel  have  been 
exited  to  the  functions  under  realistic  conditicxis.  Questions  of 
suitability  and  performance  are  typical  for  these  functicxis  and  the 
early  involvement  of  users  and  operational  testers  is  eicouraged. 

Risk  reducticxi  procedures  such  as  prototyping,  simulation,  and 
evolutionary  acquisitioi  may  also  be  appropriate. 

b.  Automation  or  ModificatiCTi  of  Existing  Capabilities;  The  transiticxi 
from  manual  functicxis  to  automated  functions  is  notoriously  hard  to 
manage.  Functions  performed  by  humans  are  usually  difficult  to 
formally  describe;  it  is  therefore  hard  to  test  the  conformance  of 
the  autOTated  cap^ility  to  a  set  of  technical  specifications.  On 
the  other  hand,  the  functions  themselves  are  gaierally  mature,  so 
suitability  in  an  operational  setting  is  not  a  critical  issue.  There 
^ould  be  a  clear  plan  for  determining  the  exteit  to  which  the 
previously  manual  capability  has  been  faithfully  reproduced. 

c.  Mature  Software;  It  is  also  possible  that  the  software 
irrplanentaticai  of  the  function  is  mature.  This  is  generally  the 
lowest  risk  iirplemoitation  of  the  function.  Many  modem  software 
design  methodologies  promote  the  reuse  of  software  as  a  way  of 
improving  overall  software  quality.  If  reusable  software  is 
included,  thei  the  TEMP  should  discuss  the  extent  to  whidi  previous 
testing  can  be  applied  to  the  current  implementation. 

References  for  Note  4t  [Redwine,  Chapter  III 
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TEMP  SectiCTi/Subsection 


Checklist  (Xiestions 


Part  I  -  Description 
2.  System. 

b,  ^tfijFfagSS:  these  are  points  _  Is  software  in^rtant  to  the 

of  interaction  with  other  interfaces?  (Note  5) 

^sterns  that  are  required  to 

acconplish  the  roissicn,  _  Do  the  interfaces  have  software 

implications?  (Note  5) 
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Electronic  interfaces  betweai  systenns  are  frequently  responsible  for 
interoperability  and  conmunicaticn.  The  impact  of  these  interfaces  on  the 
softvrare  should  be  discussed.  It  is  unlikely  that  electronic  interfaces  can 
be  designed  without  significant  software  involvement. 

For  exanple,  the  software  may  be  important  to  the  successful  inplementation 
of  the  interface.  Evai  if  the  interface  hardware  is  off-the-shelf,  the 
software  is  likely  to  be  unique  to  the  current  system  and  is  therefore  of 
higher  risk.  If  the  interface  corresponds  to  a  key  function,  thai  the 
software  should  be  treated  as  mission-critical  (cf.  Note  3). 

Further,  the  interfaces  may  place  additicmial  requirements  on  the  software. 

For  example,  coiiinunicating  systems  of  processors  frequaitly  require 
specialized  formatting  of  data  for  transmission.  Such  data  formatting 
functicxis  are  usually  software-implemented.  In  many  applications,  there  are 
industry  standards  that  can  be  invoked.  If  the  interfaces  place  requirements 
on  the  software,  these  should  be  discussed  and  (in  an  appropriate  section)  a 
plan  should  be  presented  for  ensuring  that  the  requirements  have  been  met. 

Referaices  for  NOte  5t  [Choul  ( 
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2. 


c.  Unique  Character Isticst  _ _  Does  the  system  use  software 

these  are  aspects  of  the  engineering  technology  that: 

^stem  that  make  it  better 

than  or  different  from  _  Affects  risk?  (Note  €a.) 

alternative  systems,  or 

that  lead  to  special  test  _  Has  lifecycle  impact? 

requirements.  (Note  €b) 

_  Are  there  any  software  character¬ 
istics  or  aspects  of  the  software 
application  that  distinguidi  the 
^stem  from  alternative  systems? 
(Note  6c) 


-16- 


ms...  Si 


The  toolSf  methodologies  and  engineering  techniques  that  are  used  to  produce 
operational  software  are  key  sources  of  risk  in  a  development  program. 
Software  contractors,  for  example,  frequently  conpete  on  the  basis  of 
TOftware  "methodologies"  that  conbine  management  approaches  and  technologies 
in  unique  ways.  Therefore,  some  novelty  in  the  i^stero  is  to  be  ^cp^ed. 
However,  the  TEMP  should  discuss  thc^e  novel  aspects  that  might  affect  an 
overall  evaluation. 

a.  Affect  Riskt  A  principal  source  of  risk  is  the  toolset  or  support 
SPftvarg  used  to  construct  the  operational  software.  Examples  of 
support  software  and  software  tools  range  from  any  of  the  relatively 
mature  text  editors  that  can  be  purchased  off-the-shelf  to  the 
special  purpose  conpilers  that  produce  object  code  for  the 
cperational  or  target  conputers.  The  choice  of  high  order  language 
(HOD  used  in  the  project  is  a  frequent  source  of  risk  that  may 
contaminate  the  vdiole  system.  In  particular,  new  HOL's  increase  risk 
along  several  dimensions: 

1.  a  "learning  curve"  effect  limits  the  productivity  of  design  and 
implementation  teams  during  early  project  phases, 

2.  the  immaturity  of  the  HOL  conpiler  increases  the  likelihood  that 
software  errors  may  be  introd^ed  during  the  implementation  or 
that  inefficiencies  in  compiler-generated  code  may  limiit  the 
ability  of  the  system  to  meet  performance  goals, 

3.  the  suitability  of  the  new  H(3L  for  the  applicaticxi  may  be 
undemonstrated. 

Sometimes  independent  risk  reducticxi  is  available  for  support 
software.  For  exanple,  new  Jovial  and  Ada  compilers  should  have  been 
DoD  certified  before  their  use  on  the  project,  Evaluatiai  of  support 
software  should  also  specify  what  has  not  been  assessed;  for 
instance,  performance  parameters  of  conpilers  are  not  typically 
addressed  during  compiler  validaticxi.  When  these  concerns  are 
important  to  the  overall  missicxi  of  the  system,  appropriate  T&E 
i^ould  be  planned. 

b.  Lifecycle  Impact:  A  common  source  of  operational  software  problems 
is  the  difficulty  of  maintaining  and  supporting  the  software  once  it 
is  deployed.  The  tedinology  used  to  design  and  implement  the 
TOftware  may  significantly  affect  this  ability.  Danger  signals  may 
include  the  use  of  prqprietary  tools  and  techniques  that  will  not  be 
available  to  engineers  after  system  delivery.  Alternatively,  there 
may  be  unique  aspects  of  the  design  effort  that  pesitively  affect 
subsequent  lifecycle  cost  and  effort.  One  approach  to  reducing 
long-term  lifecycle  risks  is  to  aiforce  the  use  of  common  technology 
throughout  the  development  and  operation  of  the  software.  It  is  not 
uncommon  for  the  project  office  to  supply  tools  and  support  software 
GFE  to  the  contractor  to  oisure  commonality.  However,  care  should  be 
©cercised  to  avoid  Government  liability  in  cases  of  inadequate 
Government  furnidied  tools. 


-17- 


TEMP  Sectioi/Subsection 


Checklist  (Xiestions 


Part  I  -  Description 

2. 

Does  the  system  use  software 
engineering  technology  that: 

_  Affects  risk?  (Note  fia) 

_  Has  lifecycle  inpact? 

(Note  fib) 

_  Are  there  any  software  character¬ 
istics  or  aspects  of  the  software 
application  that  distingui^  the 
system  frcan  alternative  systems? 
(Note  fic) 


c.  ttiioue  Characteristics: 
these  are  aspects  of  the 
system  that  make  it  better 
than  or  different  from 
alternative  systems,  or 
that  lead  to  special  test 
requiranents , 
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Note  6  (cont.) : 


IdeaUy,  lifecycle  characteristics  of  operational  significance  should 
be  listed  as  required  characteristics  of  the  system  (cf.  Note  7)  and 
tests  should  be  planned  to  address  the  issues  that  arise  from  these 
characteristics. 

BiStinquish  from  Other  Systems:  IVro  types  of  distinguishing 
characteristics  are  important:  those  that  differentiate  the  given 
application  fron  all  others  and  those  that  distinguish  the  curr«it 
generation  of  the  system  from  its  predecessors. 

Certain  applications  (e.g.f  those  with  nuclear  iirplications)  are 
subject  to  requirenents  and  certifications  that  are  not  levied  on 
c^er  applications.  Care  should  be  taken  to  determine  the  extent  to 
vmich  softv^re  is  r^resented  in  these  applications.  Hie  approach  to 
softv^re  testing  taken  in  some  of  these  applications  may  be 
questicxied.  The  descriptions  of  indqioidait  evaluations,  validations 
and  certifications  for  these  applicaticais  should  define  terms. 

Typical  questions  to  be  raised  are  the  following: 

Does  the  cost  of  testing  balance  the  "cost"  of  failure  in  this 
^licaticMi? 

Does  the  testing  approach  require  new  or  undemcxistrated  software 
or  hardware  technology  that  in  itself  raises  risk?  (cf .  Note  6a) 

The  system  may  also  be  unique  to  the  extent  that  the  software  is 
responsible  for  extreme  performance  or  reliability  goals.  A  proposed 
^stOT  may  raise  quantitative  requiremaits  by  one  or  more  orders  of 
magnitude.  During  early  program  phases,  clear  demcxistration  that  the 
g^ls  can  be  met  should  be  provided.  Adequate  demonstrations  can  be 
obtained  by  proof-of -concept  prototyping,  analytical  studies,  and  in 
some  instances  by  non-standard  acquisition  strategies  (e.g., 
evolutionary  or  P3I) . 

£g£efg>ce§  for  Note  6:  IRedwine,  Chapter  nil,  [step3.  Part  21 
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TEMP  Section/Subsection 


Checklist  Questions 


Part  I  -  Description 

3,  Required  Operational  Character¬ 
istics.  operational 
effectiveness  and  suitability 
characteristics,  goals  and 
thresholds. 


4,  Required  Technical  Characteristics. 
Key  technical  characteristics, 
performance  goals,  and  thresholds. 


Note:  The  characteristics  listed 
in  3-4  above  should  include,  but 
not  be  limited  to,  the  character¬ 
istics  identified  in  the  Decision 
Milestone  documentation.  Clearly 
define  these  characteristics, 
particularly  in  the  areas  of 
reliability,  availability,  and 
maintainability.  Indicate  program 
milestones  at  vihich  the  thresholds 
will  be  or  have  been  deroaistrated. 


Are  there  c^rational  or  tech¬ 
nical  characteristics  that  are: 

_  Ikiique  to  software? 

(Note  7) 

_  May  have  been  overlooked? 

(Note  7) 
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Note  7! 


A  necessary  conponent  of  system  level  test  planning  is  the  definition  of 
goals  and  thresholds  for  the  critical  software  ccnponents.  A  TEMP  for  a 
system  that  contains  missicai  critical  software  should  also  describe  the 
primary  indicators  of  the  software's; 

—  conformance  to  writtai  specifications  (required  technical 
character ist ics) 

—  operational  suitability  and  effectiveness  (required  operational 
characteristics) . 

A  TE|p>  that  fail?  tb.  define  these  characteristics  for  critical  software 
conoonoits  is  deficimt  in  that  it  has  failed  to  set  goals  and  threghniHa  fnr 
the. . characteristics  of  mission  critical  functions. 

Special  care  should  be  taken  to  ensure  that  recaiired  software  chararterlstics 
have  been  presented.  A  major  reason  for  cxnitting  references  to  software  in 
the  required  characteristics  is  that  the  software  characteristics  have 
aspects  that  are  imique  to  software  technology.  The  framer  of  the  TEMP  may 
have  little  experimce  in  judging  the  relative  inportance  of  these 
characteristics, 

Aiother  reason  for  not  including  software  characteristics  in  the  TEMP  is  that 
they  do  not  fit  cleanly  into  the  technical/operational  definitions.  In  factr 
OTe  distinguishing  feature  of  software  is  that  the  goals  and  thresholds  of 
interest  may  blur  the  distinction  between  technical  and  operational 
characteristics.  In  develqping  test  plans  and  schedules /  care  must  be 
exercised  to  ensure  that  software  characteristics  are  evaluat-pd  af  j-he^ 
3RPJ.PPri9te.  Stage  of  system  develoment  rather  than  at  arbitrarily  imposed 
IBAlestbnes*  it  is  a  mistake  to  wait  until  the  hardware  and  software  are 
integrated  to  resolve  outstanding  software  test  issues  (cf .  Note  8) .  For 
instance,  some  operaticml  parameters  associated  with  the  software  can  be 
reliably  determined  during  developmait  testing  txjt  cannot  be  measured 
directly  during  operational  testing. 

Early  evaluaticxi  of  software  characteristics  should  be  an  integral  part  of 
the  development  process.  Late  treatment  of  the  software  opens  the  following 
problans: 

grror  maskingt  hardware  and  software  errors  may  in  some  instances 
mask  each  other,  complicating  RAM  analysis. 

error  partitioning;  without  a  reliable  estimate  of  software  failure 
rates,  the  partiticxiing  of  operational  errors/ failures  into  hardware, 
operator,  and  software  failures  may  be  subjective  and  inexact. 
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Egg.  giep.ti<av!g.ubsfi.gfclgn 


Checklist  Oiestions 


Part  I  -  Descript iCTi 

3.  Required  Oueraticml  Character¬ 
istics.  Key  operational 
effectiveness  and  suitability 
characteristics,  goals  and 
thresholds. 


4.  Required  Technical  Characteristics. 
Key  technical  characteristics, 
performance  goals,  and  thresholds. 


Note:  The  characteristics  listed 
in  3-4  above  should  include,  but 
not  be  limited  to,  the  character¬ 
istics  identified  in  the  Decision 
Milestone  documentation.  Clearly 
define  these  characteristics, 
particularly  in  the  areas  of 
reliability,  availability,  and 
maintainability.  Indicate  program 
milestones  at  which  the  thresholds 
will  be  or  have  been  demaistrated. 


Are  there  operational  or  tech¬ 
nical  characteristics  that  are: 

_ _  Iftiigue  to  software? 

(Note  7) 

_  May  have  beoi  overlooked? 

(Note  7) 


-22- 


Note  7  (cont.): 


As  a  guide  to  locating  and  evaluating  required  software  characteristics,  the 
following  examples  should  be  taken  into  account. 

1.  Beliabilityi  This  characteristic  is  often  a  key  indicator  of 
software  suitability.  It  is  very  iirportant  to  choose  metrics  and 
measurerooit  criteria  that  adequately  reflect  software  reliability. 

Qi  the  other  hand,  it  should  be  recognized  that  software  reliability 
has  unique  a^>ects.  Classical  time-dependait  reliability  theory  may 
not  apply  to ‘software.  Probability  distributions  are  notoriously 
ineffective  in  describing  failure  rates  for  software  exc^  in  cases 
vrtiere  tte  true  operational  distributions  of  the  inputs  are  known. 
Qaserving  software  failures  in  integrated  hardware/software  ^sterns 
is  difficult.  Low  reliability  estimates  for  software  that  inplements 
critical  fuMticxis  should  be  questioned.  If  the  software  is 
duplicated  in  multiple  platforms,  then  failure  rates  are 
multiplicative  (since  many  instructiai  executicns  take  place) ,  and 
even  very  low  failure  rates  Ccin  result  in  significantly  many 
op®i^3tional  roissi(xi  failures.  The  use  of  tests  that  exercise  and 
stress  software  conponents  and  demonstrate  functional  behavior  in 
simulated  operational  environments  should  be  considered. 

^Silabilitv  and  Maintainability t  Bardware-oriented  definitions  of 
availability  and  maintainability  are  seldom  satisfactory  for 
software.  Parameters  such  as  mean  logistic  down  time  that  take  into 
account  spare  parts  requirem^ts  and  transportation  time  do  not 
adequately  cepture  software  availability.  Maintainability  of 
^ftware  incorporates  repair  and  re-engineering.  Usually  maintoiance 
is  carried  out  at  a  Post  D^loyment  Software  Support  (PDSS)  facility 
and  fartors  limiting  mean  time  to  repair  taid  to  revolve  around 
conmunications  and  the  labor-intensivoiess  of  the  maintenance 
process. 

3.  Human  Factors;  As  an  indicator  of  operational  suitability,  this 
aspect  can  be  evaluated  early.  Bie  use  of  simulators,  prototype 
hardware,  and  operator  persoinel  can  give  reliable  indications  of 
software  suitability  in  the  laboratory  setting.  Early  determination 
of  deficiencies  allows  correction  through  redesign  of  the  software. 
Late  detection  of  unsuitable  human  factors  in  the  softvare  can  raise 
the  cost  of  correcticn  by  one  or  more  orders  of  magnitude. 

4.  Pfirformanco:  operational  performance  parameters  may  be 

determined  very  early  in  the  development  process  by  technical 
software  characteristics.  For  exairple,  the  ability  of  a  system  to 
track  and  engage  multiple  targets  may  be  limited  by  the  precision  and 
accuracy  of  the  algorithms  used  in  software  design,  the  efficieicy  of 
a  frequaitly  executed  mathematical  subroutine,  or  by  the  size  of  a 
software  buffer.  Another  performance  threshold  may  be  the  ability  of 
the  lystem  to  operate  in  a  degraded  mode  above  a  certain  threshold. 
Such  a  performance  parameter  may  be  solely  due  to  the  "robustness"  of 
the  software,  a  technical  characteristic  that  indicates  how  well  the 
software  operates  vAien  its  input  does  not  satisfy  the  input 
specifications . 
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3 


istics.  operaticxial 
effectiveness  and  suitability 
characteristics f  goals  and 
thresholds. 


4. 


lired  Technical  Characteristics. 


Key  technical  characteristics, 
performance  goals,  and  thresholds. 


Note:  The  characteristics  listed  _  Are  there  operaticml  or  tech- 

in  3-4  above  should  include,  but  nical  characteristics  that  are: 

not  be  limited  to,  the  character¬ 
istics  identified  in  the  Decision  _  Dhigue  to  software? 

Milestone  documentaticxi.  Clearly  (Note  7) 

define  these  characteristics, 

particularly  in  the  areas  of  _  May  have  beai  overlooked? 

reliability,  availability,  and  (Note  7) 

maintainability.  Indicate  program 
milestones  at  which  the  thresholds 


will  be  or  have  been  danonstrated. 
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Note  7  (cont.) : 

In  summary,  "performance"  —  as  the  term  is  commonly  applied  to 
software  systems  —  includes  the  quantified  efficiency  or  capaci^  of 
the  progreans.  Evai  though  the  primary  (^stem-level)  characteristic 
is  operational,  the  best  predictor  of  software  performance  is  usually 
technical. 

Inferences  for  Note  It  [AFOTECIII],  [IBM],  [Meyers],  [Perils] 
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TEMP  Section/Subsection 


CtiegKlist  ..CMgstions 


Part  I  -  Description 

5*  -Critical  T&E  Issues  _  Do  the  required  software  charac¬ 

teristics  raise  unique  or  easily 

a.  Technical  Issues;  over  lodged  T&E  issues?  (Rote  8) 

areas  of  technological  or 

engineering  risk  that  must  be 
addressed  by  testing. 

b.  Operational  Issues;  k^ 
operational  effectiveness  or 
suitability  issues  that  must 
be  addressed  by  testing. 
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I32te_a: 


A  critical  software  T&E  issue  is  any  aspect  of  the  software  system's 
capabili^  that  must  be  quest iaied  before  a  system's  overall  worth  can  be 
estimated.  The  software  issues  are  of  primary  inportance  in  reaching  a 
decisicxi  as  to  whether  the  system  should  advance  into  later  programmatic 
phases.  This  decision  is  to  be  based  in  part  on  the  determinatioi  that  the 
goals  and  thresholds  defined  for  the  required  software  characteristics  have 
been  met  and,  in  any  case,  should  be  based  on  an  assessment  that  software  and 
hardware  risk  have  been  balanced  by  the  past  and  future  T&E. 

In  addition  to  other  discussions  of  software  issues  that  may  be  present  in 
the  TEMP,  goals  and  thresholds  should  be  associated  with  each  issue  that  will 
be  addressed  by  testing.  This  will  be  the  basis  for  judging  the 
effectiveness  of  more  detailed  software  test  plans  and  will  provide  the 
framework  for  interpretation  of  software  test  results. 

For  example,  a  critical  software  T&E  issue  may  be  that  the  maintainability  of 
the  software  is  to  be  validated  (cf .  Note  7) .  Maintenance  of  operational 
software  may  require; 

1.  a  PDSS  or  similar  support  facility  that  is  adequate  for  the 
re-engineering  effort  that  may  be  required  during  maintoiance 

2.  a  logistics  support  network  for  transferring  maintained  software  from 
the  PDSS  to  the  fielded  system 

3.  the  skilled  human  resources  necessary  to  re-engineer  a  large  software 
^stem  under  severe  scheduling  constraints. 

The  validation  of  software  maintainability  as  an  operational  parameter  (e.g., 
is  the  MPTR  for  critical  software  caiponents  sufficiently  small  to  ensure 
that  system  availcdDility  goals  can  be  met?)  leads  to  the  following  question: 
is  the  test  environment  representative  of  the  environment  in  which  the 
software  will  actually  be  maintained?  The  T&E  outlines  should  provide  a 
detailed  answer  to  this  question.  In  the  case  of  validating  software 
maintainability,  the  PDSS  personnel  should  conduct  the  tests.  Use  of  PDSS 
personnel  meets  one  objective  of  an  operational  test  —  use  of  typical 
operator  personnel  in  a  typical  operational  environmant  —  and  also  ensures  a 
realistic  estimation  of  the  maintainability  characteristic. 

As  ^inted  out  in  Note  7,  quantifiable  progress  toward  goals  is  the  most 
desirable  way  of  posing  a  critical  issue.  However,  objective  evaluations  of 
progress  are  oftentimes  more  important  than  ad  hoc  quantification.  For 
example,  meeting  a  time  depoideit  reliability  goal  R(t)  =  0.97  for  t  hours  of 
operation  is  seldom  meaningful  for  software,  and  any  attempts  to  provide  such 
a  statistical  measure  should  be  questioned.  On  the  other  hand,  achieving  an 
observed  mean  time  betweoi  operatiaial  mission  failure  (MTBCttlF)  of  t  hours  is 
meaningful  and  allows  analysts  to  concentrate  on  validating  the  realism  of 
the  test  scaiario,  the  software  contributiai  to  observed  operatimal 
failures,  and  other  issues  that  help  determine  the  indices  of  progress  for 
the  software. 

As  further  discussed  in  Note  7,  issues  should  also  be  defined  to  espose 
software  uniqueness  in  the  issues  and  should  associate  technical  or 
operational  meanings  to  the  issues,  regardless  of  the  standard  (hardware) 
interpretaticai. 

References  for  Note  8;  [Brown],  [STEPl] 
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TEMP  Sgctipp/SutPSegtiQP  Checklist  Questions 


E^.rt_  II  -  Program  Sumtary 

1.  Management.  Outline  the  program 
and  T&E  roanagerooit  responsibili¬ 
ties  of  participating  organiza¬ 
tions.  Highlight  arrangemaits 
between  participants  for  test 
data  sharing,  responsibilities 
for  test  managaneit  decisions, 
and  management  interfaces  for 
multiservice  T&E  efforts.  Dis¬ 
cuss  the  adequacy  of  the  planned 


Is  there  a  manager  with  principal 
responsibility  for  software  in 
the  project  office?  In  test 
organizati(»)s?  (Note  9) 

Are  the  project  offices  and  test 
organizations  aware  of  T&E  that 
will  be  carried  out  ly  parallel 
organizaticxis?  (Note  10) 


test  periods  and  schedule  to  pro-  _  Are  the  results  of  tests  of  soft- 

vide  confidence  in  test  results.  ware  conponaits  available  to  sub- 

sequQit  test  groups?  (Note  10) 
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Note  9; 


Experience  has  shown  that  all  aspects  of  software  developmoit  and  testing 
progress  more  efficiently  when  a  knowledgable  softvrare  manager  is  presoit  and 
active.  Software  issues  tend  to  cut  across  system  issues.  The  program 
managemoit  structure  should  aisure  that  software  concerns  are  not  left 
unattended. 

Peferaices  for  Note  9:  [Brown],  [STEPS,  Part  2] 


Note  lOt 

A  nunber  of  DoD  and  non-DoD  organizations  carry  out  testing  and  validaticai 
for  both  operaticml  and  support  software  (cf .  Note  6a) .  These  organizations 
and  the  evaluaticxis  they  carry  out  include  the  following: 

—  DoD  Security  Ceiter  (software  security  certificatiai) 

—  Natiaial  Bureau  of  Standards  (software  cryptologic  certification  for 
Data  Encryption  Standard) 

—  Federal  Software  Testing  Caiter  (testing  of  coitpilers  and  support 
software  for  conformance  to  specifications) 

—  Ada  Joint  Program  Office,  OUSDRE  (validation  and  certification  of  Ada 
conpilers) 

—  Air  Force  Language  Coitrol  Facility  (validation  and  certification  of 
Jovial  compilers,  cataloging  of  tools  for  Jovial  programming 
eivironments) 

—  Product  Engineering  Services  Office,  OUSERE  (evaluation  of  FOT&E  for 
^sterns  in  production) 

In  addition  several  groups  of  testers  may  proceed  independently.  Contractors 
may  produce  test  results  that  are  useful  to  indepoident  Government  testers. 
Developmait  testers  may  generate  results  of  simulations  that  provide 
indicaticxis  of  cperational  effectivaiess  to  cperational  testers.  Operational 
test  scaiarios  may  be  analyzed  by  developmeit  testers  to  determine  software 
test  coverage.  In  each  applicable  instance,  possibilities  for  test  data 
sharing  and  incorporatiai  of  independent  certificaticxis  into  the  TEMP  should 
be  questioned. 

References  for  Note  10 t  [STEPS,  Part  2] 
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1.  Mangggngpt.  Outline  the  program  _  Is  there  evidence  that  available 

and  T&E  management  responsibili-  sources  of  ejspertise  have  been 

ties  of  participating  organiza-  explored  and  that  coordination 

ticns.  Highlight  arrangements  has  been  carried  out  with 

betweoi  participants  for  test  programs  designed  to  assist  MCX31 

data  sharing,  responsibilities  software  projects?  (Etote  11) 

for  test  managanait  decisions, 

and  management  interfaces  for  _  Are  management  aids  (e.g.,  tools, 

multiservice  T&E  efforts,  Dis-  checklists,  guides,  and  decision 

cuss  the  adequacy  of  the  planned  support  systems  being  used? 

test  periods  and  schedule  to  pro-  _  (Note  12) 

vide  confidence  in  test  results. 

_  Is  there  a  plan  to  use  electronic 

mail/ccmmunicaticxis  among 
participating  organizations? 

(Note  12) 
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lit 


Bie  effective  utilization  of  existing  softvrare  technology,  practices,  and 
management  techniques  may  be  aided  by  the  assistance  of  special  programs  in 
DoD  and  the  military  Services. 

Overall  responsibility  for  coordinating  software  initiatives  and  technology 
transition  programs  in  DoD  rests  with  the  Director,  Ccxiputer  Software  and 
Systems  (OUSERE) .  These  programs  may  offer  human  and  technical  resources  to 
assist  MOCR  software  development  efforts. 

Army,  Ifevy,  and  Air  Force  focal  points  for  software  programs  vary.  However, 
the  Joint  logistics  Commanders  (JLC)  have  established  a  computer  resource 
managemait  board  and  each  of  the  Services  has  appointed  a  Computer  Resources 
Manager  (CI91) .  Bie  CRMs  are  sources  of  information  concerning 
Service-specific  programs  and  initiatives. 

Sources  of  e^^ertise  in  software  matters  related  to  software  testing,  during 
all  phases,  in  DOD  are  concaitrated  in  the  Office  of  the  Director  Defuse 
Test  and  Evaluation. 

Within  the  Services,  the  Development  Test  Commanders  and  Operatiaial  Test 
Commanders  are  the  appropriate  contacts  for  information  concerning  software 
testing  resources. 

Program  roanagemoit  descriptions  shoxild  moition  any  such  supporting  ao±ivities 
or  clearly  indicate  that  no  additional  sources  of  expertise  are  needed  during 
the  current  program  phase. 

References  for  Note  lit  [Klucas] ,  [SrEP2,  Appendix  A] 


12: 

Over  the  past  several  years,  the  technolojgy  to  support  management  of  software 
projects  has  improved  rapidly.  Automated  tools  are  available  to  estimate  and 
track  project  co>sts,  sodiedule  tasks  and  monitor  their  pro)gress,  and  implemoit 
a  number  of  other  management  functions.  In  addition,  checklists  and  manuals 
such  as  this  one  have  beoi  developed  for  other  aspects  of  software 
acquisition.  Finally,  automated  decision  support  systemis  with  accompanying 
databases,  local  area  networks,  wide-band  communication  capabilities, 
workstations,  and  tools  for  supporting  acjquisitioi  decision-making  are 
available  commercially. 

Software  managers  in  project  offices  and  test  organizaticxis  should  be  aware 
of  the  available  technolojgy  and  should  have  made  a  cost-benefit  assessment  of 
the  desirability  of  using  such  technology  (cf .  Note  11) . 

Refer^ces  for  Note  12;  [AFOTECIl ,  [Wattl 
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TEMP  Section/Subsection 


Checklist  Questions 


Program  gwp&ry 

2.  Integrated  Schedule;  Di^lay . 
the  integrated  time  sequen¬ 
cing  of  T&E  for  the  oitire 
program  and  related  key  evaits 
in  the  acquisition  decision¬ 
making  process.  Include  such 
events  as  program  decision 
milestones/  key  subsystem 
demonstrations,  test  article 
availability,  first  flights, 
critical  si^jport  resource 
availability,  critical  full-up 
system  demcnst  rat  ions,  key 
OT&E  events,  first  production 
deliveries,  and  initial 
operational  capability  date. 


Are  k^  software  subsystem 
demonstratiais  included  in  the 
integrated  schedule?  (Note  13) 

Does  the  schedule  show  software 
deliveries  and  tool  availability 
dates?  (Note  14) 
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Ihe  integrated  schedule  should  include  such  events  as  key  software  subi^stem 
demonstraticMis  and  software  test  article  availability.  The  schedule  should 
also  include  adequate  allowance  for  repair  and  retest  of  software^  as  well  as 
time  to  perform  the  original  tests.  Failure  to  do  so  indicates  ttet  proper 
planning  for  critical  software  conponents  has  not  tak^  place. 

References  for  Note  13 t  [STEPl] 


Note  14; 

Support  resource  availability  should  be  displayed  in  the  integrated  schedule. 
Software  testing  tools  fall  into  this  category  and  deserve  ^)ecial  mention. 
Since  these  tools  are  themselves  softwarer  their  developnent  and  acquisiticn 
are  subject  to  many  of  the  same  risks  as  any  other  software  develcpmait  (cf. 
Note  6a) .  The  availability  dates  of  these  tools  should  be  included  in  the 
TEMP  and/or  tracked  carefully  by  other  means  since  a  late  delivery  could 
impact  the  entire  system  developnent  effort. 

References  for  Note  14;  [STEPll,  ISTEP2,  Part  31) 


TEMP  Sectioi/Subsection 


Checklist  CXiestions 


m  -  PTS.E..Qtftline 

This  outline  should  discuss  all  ET&E 
in  sufficient  detail  so  that  test 
objectives  are  related  to  the  system 
operational  concept  and  are  clearly 
idsitified  for  each  phase.  Relate  the 
planned  testing  to  the  critical  • 
technical  issues  appropriate  to  each 
phase.  Hie  following  information 
^ould  be  included: 

1.  DT&E  to  Date.  A  summary  of  DT&E 
already  conducted  based  on  the 
best  available  information. 

Briefly  describe  test  articles 
with  enphasis  on  how  they  differ 
from  planned  production  articles. 
Emphasize  DT&E  events  and  results 
related  to  required  performance 
characteristics,  critical  issues, 
and  requirements  levied  by  earlier 
OSD  decisions.  Highlight 
technical  characteristics  or 
specification  requirements  that 
were  denoistrated  (or  failed  to 
be  demaistrated) .  Describe  how 
simulation  models  were  validated. 


Have  cperational  characteristics 
of  the  software  that  can  be 
demonstrated  during  DT&E  been 
identified?  (Note  7) 

Is  there  a  plan  for  demonstrating 
appropriate  operational 
characteristics  of  the  software 
during  each  phase?  (Note  7) 

Have  planned  levels  of  testing 
been  achieved?  (Note  15) 

Is  the  documaitaticm  that  r^rts 
software  test  results  cited? 

(Note  15) 
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Note  15t 


It  should  be  apparent  from  the  description  of  the  software  tests  conducted 
and  their  results  whether  or  not  previous  goals  have  been  met  and  test 
objectives  have  been  satisfied.  Vague  referaices  to  "successful  software 
tests"  or  "no  problems  with  the  software"  should  not  be  acceptable.  In  order 
to  evaluate  the  progress  of  software  testing  to  date,  there  roust  be  explicit 
referaice  to: 

—  a  systematic,  scientifically  sound  approach  to  carrying  out  the  test 

—  the  relaticxiship  between  the  systematic  test  approach  and  the  test 
objectives  for  the  current  phase 

—  the  results  of  the  test 

—  the  plans  for  resolution  of  errors. 

For  example,  during  very  early  program  phases,  unit  and  module  testing  will 
be  conducted  by  contractors  under  government  planning.  The  results  of  these 
tests  should  be  maintained  by  the  contractor .  The  applicable  Military 
Standards  may  specify  the  content  of  these  test  results.  If  no  format  is 
contractually  specified,  it  may  be  desirable  to  inquire  into  methods  wherdDy 
test  results  can  be  sunmarized,  archived,  audited,  and  conntunicated  to  test 
organizations  conducting  higher  level  tests  (cf ,  Note  10) . 

Systematic  tests  at  this  stage  can  provide  indications  of  progress  toward 
solving  such  issues  as  operaticml  suitability  (e.g.,  suitability  of  user 
interfaces  and  coverage  of  software  ^stem  requirements) .  100%  statement 
coverage,  cooplete  functional  tests,  or  randcro  tests  with  specified  input 
distributiais,  durations,  and  ccxifidence  limits  are  examples  of  i^stematic 
test  approaches  that  can  be  carried  out  to  support  these  objectives. 

Higher  level  tests  may  cite  other  test  approaches  or  the  compositicn  of  lower 
level  tests,  menticxi  results  of  simulations,  ^)eci^  goals  for  continuous 
operation  under  varying  load,  and  define  a^roaches  to  loading  or  stressing 
software  that  demonstrate  the  performance  limits  of  critical  software 
functicais.  In  all  of  these  cases,  however,  tests  should  not  be  considered 
performed  and  software  issues  should  not  be  considered  resolved  unless  the 
TEMP  reviewer  is  convinced  that  the  test  methodologies  cited  have  been 
carried  out  to  conpletion  and  that  the  results  of  the  tests  are  available  for 
examination,  applicable  Military  Standards  refer  to  Data  Item  Descrip>ticxis 
(DID's)  v^ich  specify  the  format  and  conteits  of  higher  level  test  results, 

Kefergices  for  Note  15;  [srEP2,  Part  1],  [STEPS,  Part  31,  [STEPS,  Bowen] 
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3^  Sgcticn/S»l?sectipn 


Checklist  Oiestions 


Pgrt.  Ill  -  TOE  Qtftline 

1,  DT&E  to  Date.  A  sumraary  of  DT&E 
already  conducted  based  on  the 
best  available  information. 

Briefly  describe  test  articles 
with  emphasis  on  how  they  differ 
from  planned  production  articles. 
Emphasize  DT&E  evoits  and  results 
related  to  required  performance 
characteristics,  critical  issues, 
and  requirements  levied  by  earlier 
OSD  decisions.  Highlight 
technical  characteristics  or 
specification  requirements  that 
were  danaistrated  (or  failed  to 
be  demcxTStrated) .  Describe  how 
simulation  models  were  validated. 


Have  software  deficiaicies 
revealed  by  DT&E  been  interpreted 
in  terms  of  required  system 
characteristics  and  critical 
issues?  (Note  16) 

Is  the  evidence  clear  that  hard¬ 
ware  and  software  failures  have 
been  properly  partitioned? 

(Note  16) 

Have  demoistrated  software 
characteristics  been  highlighted? 

(Note  16) 
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Software  technology  is  notorious  for  its  jargon.  Jargm  is  especially 
difficult  to  interpret  in  test  r^Jorts.  Phrases  like  "abend  at  locatiai 
11324"  and  "buffer  overflow  causing  module  JXAS115  to  hang"  describe  test 
evaits  very  precisely  and  may  be  helpful  to  software  oigineers  engaged  in 
error  location  and  removal  —  however r  these  phrases  are  not  very  meaningful 
to  system  engineers.  Rather  than  camouflage  software  deficiaicies  with 
overly  technical  descripticxis,  software  DT&E  TEMP  descriptions  should 
concentrate  on  technical  goals,  thresholds,  and  c*)jectives.  At  each  review 
phase,  the  essential  questions  should  continue  to  be:  Were  the  ET&E 
objectives  met?  If  they  were,  with  vdat  degree  of  confidence  were  they  met? 
If  they  were  not  met,  vrtiat  was  the  specific  bdiavior  that  led  to  the  observed 
anomaly? 

IXiring  operatiaial  tests,  the  relaticaiships  betweai  test  events,  software 
deficiaicies,  and  unresolved  test  issues  are  more  difficult  to  discover.  In 
fact,  the  recording  of  test  results  in  the  operaticaial  setting  may  not  be 
adequate  for  reccxistructing  the  cause  of  a  particular  software  failure, 
therefore,  a  failing  during  OT&E  is  not  the  overly  technical  descripticms  of 
software  failures,  but  is  rather  the  teidency  to  note  a  software  anomaly  with 
so  little  suH»rting  information  that  traceability  to  critical  operational 
T&E  issues  is  not  feasible  (cf .  Note  20) . 

Relating  software  T&E  results  to  system-oriented  T&E  issues  helps  to  ensure 
that  responsibilities  for  deficiencies  are  prc^rly  allocated  between 
hardware  and  software.  The  TEMP  should  provide  evidaice  that  this 
partitioning  of  errors  has  beai  the  result  of  conpetoit  analysis.  Claims 
that  errors  have  been  traced  to  either  hardware  or  software  should  be 
dismissed  unless  supporting  argumaits  can  be  supplied.  For  example,  a 
processor  chip  may  fail  in  an  avionics  system.  However,  if  the  software  is 
supposed  to  be  fault-tolerant,  the  error  should  probably  be  charged  to  the 
softvare  as  well  as  the  hardware.  Furthermore,  the  critical  T&E  issues 
should  address  such  instances  of  dual  responsibility. 

References  for  Note  16:  [Brown],  [STEPS,  Blackledgel 


-37- 


TEMP  Segtiop/Svibsecti-on 


Checklist  OuestiOTis 


S&Jl-J.II.  -  PK.P.-(Xitl3Jie 

1.  PT&E  to  Date.  A  summary  of  DT&E 
already  conducted  based  on  the 
best  available  information. 

Briefly  describe  test  articles 
with  arphasis  on  how  they  differ 
from  planned  production  articles. 
Biphasize  DT&E  events  and  results 
related  to  required  performance 
characteristics,  critical  issues, 
and  requiremaits  levied  by  earlier 
OSD  decisims.  Highlight 
technical  characteristics  or 
specification  requirements  that 
were  dancaistrated  (or  failed  to 
be  demoistrated) .  Describe  how 
simulatiai  models  were  validated. 


Have  the  differeices  betweai  the 
softvare  tested  and  the  planned 
operational  software  been 
emphasized?  (Note  17) 

Whs  the  test  aivironment  (e.g., 
developmait,  operational,  or 
■maintenance)  appropriate  for  the 
characteristics  to  be 
demonstrated?  (Note  17) 
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mtfi  17: 


It  is  not  unusual  to  find  significant  differences  between  the  software  during 
early  stages  of  develc^ent  and  the  software  that  will  eventually  be 
deployed.  In  extreme  cases,  the  programming  language  may  evai  change.  More 
corrmcxi  are  the  many  provisions  that  are  useful  for  conducting  such  tests  as 
unit  and  module  tests,  software  integraticxi  tests,  and  software  system 
function  tests.  These  include  the  following: 

—  Sest  Drivers  or  harnesses  to  simulate  programs  that  control  and  feed 
•  data  to  the  software  being  tested  / 

—  Snulators  to  simulate  the  instructions  of  the  operational  hardware  or 
target  computer  in  the  development  environment  or  host  conputer 

—  Simulators  for  stimulating  software  inputs  with  realistic  signals. 

Insofar  as  the  tests  that  use  these  techniques  may  be  required  to  provide 
adequate  verification  of  designs,  the  test  results  are  valuedDle.  However, 
care  should  be  taken  to  track  the  course  of  the  tested  software  between  the 
current  test  phase  and  its  integraticxi  into  system  hardware.  Any  changes 
(e.g.,  rqplacement  of  harnesses  ty  operational  software)  that  alter  basic 
functional  characteristics  will  require  retesting  at  a  later  date.  In  many 
instances,  comparison  of  these  later  tests  with  the  current  tests  will  be 
useful  for  locating  actual  differences,  so  arrangements  for  archiving  the 
test  activity  (or  reconstructing  it)  should  have  been  made. 

A  parallel  consideration  is  the  nature  of  the  test  environment.  While 
questiaiing  the  fidelity  of  siraulatiais  and  the  performance  implications  of 
target  hardware  emulations  may  be  useful  in  other  portions  of  the  TEMP,  the 
same  i^ues  ajply  to  software.  In  addition,  the  software  environment  may 
have  significant  impact  m  the  interpretation  of  the  test  results.  For 
ecanple,  the  assessment  of  whether  maintainability  goals  have  beoi  met  is 
complicated  if  the  test  environmait  does  not  correspond  to  the  environm^t  in 
vhich  the  software  will  be  maintained  (cf .  Note  8) .  Similarly,  other 
software  characteristics  may  be  influenced  by  even  minor  changes  in  the 
Qivironment.  As  noted  above,  there  should  be  capabilities  for  revisiting  the 
softvare  test  issues  addressed  in  the  currait  phase  of  testing. 

Ee^erences  for  Note  17:  [STEP2,  Part  3],  [STEPS,  Blackledgel 


TEMP  Section/Subsection 


Checklist  Questions 


Part  III  -  DT&E  Outline 

2.  Future  DTSE.  Discuss  all  remain¬ 
ing  DT&E  planned,  beginning  with 
the  date  of  the  current  TEMP 
revision.  Address  s^rately 
each  remaining  phase  of  DT&E, 
including  the  following  for  each: 

a.  Equicinait  Description:  Sum¬ 
mary  of  functiaial  capability 
and  how  it  is  expected  to 
differ  ^f rom  the  production 
model. 

b.  ET&E  Objectives:  Suitmary  of 
the  specific  DT&E  dsjectives 
to  be  addressed  during  each 
jAiase.  The  objectives  identi¬ 
fied  should  be  the  discrete 
major  goals  of  the  DT&E  effort 
v^ich,  whai  achieved,  will 
provide  soluticxis  to  critical 
technical  issues  and  demon¬ 
strate  that  the  aigineering 
effort  is  progressing 
satisfactorily.  If  the  OSD 
decisioi  memorandum  requires 
demonstraticm  of  specific 
technical  characteristics  in 

a  given  phase,  id^tify  those 
characteristics . 

c.  DT&E  Events/Scope  of  Testing/ 
Basic  Scenarios:  Key  DT&E 
evaits  planned  to  address  the 
objectives.  In  addition, 
describe  in  sufficient  detail 
the  scope  of  testing  and  basic 
test  scenarios  so  that  the 
relaticxiship  betweei  the  test¬ 
ing  and  the  c*)jectives,  and 
the  amount  and  thoroughness  of 
testing  are  clearly  apparent. 
Discuss  RAM  testing  and  define 
terms. 

3.  Critical  DT&E  Items.  All  items 
the  availabilily  of  v^ich  are 
critical  to  the  conduct  of 
adequate  DT&E  prior  to  the  next 
decision  point. 


Have  differaices  betweai  the 
software  to  be  tested  and  the 
planned  operational  software  been 
summarized?  (Note  17) 


Are  software  DT&E  test  objectives 
traceable  to  required  software 
characteristics  and  critical  T&E 
issues?  (Note  7,  Note  8) 


_  Will  the  planned  software  testing 

demonstrate  the  required  charac¬ 
teristics?  (Note  18) 


_  Are  any  new  software  subsystems 

needed  for  DT&E  prior  to  the  n^t 
decisicxi  point?  (Note  13) 

_  Are  any  new  software  support 

i^steros  or  tools  needed  for  ET&E 
prior  to  the  next  decision  point? 

(Note  14) 
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There  should  be  a  clear  relatiaiship  between  the  test  deject  ives  and  the 
softvare  tests  that  are  planned  (cf.  Note  15),  Testing  by  "bulk"  is  seldcan 
effective  for  software,  Ch  the  other  handf  statistical  methods  that  attempt 
to  predict  the  amount  of  software  testing  required  are  error-prone  and  their 
use  should  be  carefully  ^amined, 

Oiantitative,  time-dependent  goals  (e,g,,  a  fixed  mean  time  betweai  software 
failures)  are  difficult  to  demonstrate  during  ET&E,  If  quantitative 
information  is  unavailable  during  early  phases,  a  second  choice  is  to 
associate  qualitative  characteristics  with  quantitative  goals.  For  example, 
knowing  that  a  software  reliability  requirement  is  extremely  high  is  usually 
more  inportant  than  knowing  that  the  goal  is  R(t)=0,997,  In  this  case,  tests 
should  ensure  that  every  software  instruction  has  been  executed,  that  all 
likely  logic  paths  have  beoi  tested,  that  a  strategy  has  been  adopted  for 
determining  that  fatal  coding  defects  do  not  remain,  and  that  all  required 
software  functions  have  been  demonstrated.  Since  identifying  and  isolating 
logic  paths  is  ejpaisive,  the  adoption  of  such  a  test  should  be  reserved  for 
software  componoits  in  which  reliability  is  a  critical  issue.  On  the  other 
hand,  requirements  may  iirply  that  the  correct  functicxiing  of  the  software  is 
so  critical  that  very  sensitive  tests  are  needed.  In  these  cases,  the  nature 
of  the  test  and  its  relaticxjship  to  the  objective  should  be  clearly 
justified. 

Expense  is  by  itself  not  a  useful  parameter  in  judging  the  effectiveness  of  a 
test  for  demcxist rating  a  particular  characteristic.  For  instance,  many 
projects  include  a  requirement  for  software  aidurance  tests  such  as  25  hours 
of  continuous  operation.  Such  test  are  expensive,  but  they  are  seldom 
effective  in  uncovering  software  defects,  Pandcmi  sampling  of  software  inputs 
is  relatively  inexpensive.  However,  whai  sampling  distributions  are  known 
with  a  high  level  of  confidaice,  results  of  random  tests  are  good  estimators 
of  operaticxial  reliability. 

In  contrast  to  DT&E,  quantitative  time-d^endoit  measurements  are  a  principal 
result  of  OT&E,  Even  so,  special  care  must  be  exercised  to  ensure  that 
measuremaits  properly  reflect  the  software's  (peraticml  suitability  and 
effectiveness.  Care  must  also  be  exercised  when  determining  whether  or  not 
the  software's  contributions  to  overall  i^stem  RAM  and  performance 
measurements  have  beoii  adequately  r^resaited.  For  acample,  tiiis 
determination  is  dependait  upon  the  existeice  of  eiough  visibility  into  the 
software  during  OT&E  to  eisure  that  the  software  contribution  to  OT  events 
can  be  accurately  assessed. 

In  most  cases,  the  exact  nature  of  the  software  test  will  not  be  apparent  in 
the  TEMP,  The  TEMP  evaluator  should  be  prepared  to  acquire  viiatever  degree 
of  detail  is  necessary  to  determine  whether  a  given  test  objective  can  be 

met,  A  guiding  principle,  however,  should  be  that  ad  hoc,  unsystematic  tests 
are  usually  not  effective. 

References  for  Note  18 t  [Adrionl,  tSTEP2,  Part  21,  [STEP5,  Bowenl 


-41- 


This  outline  should  discuss  all  OT&E 
from  the  earliest  lOTStE  through  the 
EOTStE  during  initial  production  and 
deplcpnent.  Test  objectives  should 
relate  the  planned  testing  to  the 
critical  operaticxial  issues.  Defi- 
cioicies  in  the  productiai  system 
should  be  identified.  The  following 
information  should  be  included  in 
similar  format  and  detail  as  in  the 
DT&E  outline  (Part  III): 

1-  OT&E  to  Date.  A  summary  of  OT&E 
already  conducted  based  on  the 
best  available  information. 

Briefly  describe  test  articles 
with  emphasis  on  how  they  differ 
from  planned  production  articles. 
Brphasize  OT&E  evoits  and  results 
related  to  required  performance 
characteristics,  critical  issues, 
and  requiremaits  levied  by  earlier 
OSD  decisiois.  Relate  the  test 
conditions  and  results  to  the 
operational  effectiveness  and 
suitability  of  the  system  being 
acquired. 


_  Have  software  issues  left 

unresolved  during  DT&E  been 
addressed?  (Note  19) 

_  Has  ET&E  of  operational 

characteristics  been  related  to 
operational  goals?  (Note  19) 

_  Have  software  deficieicies 

revealed  by  OT&E  been  interpreted 
in  terms  of  required  system 
characteristics  and  critical 
issues?  (Note  16) 

_  Is  the  evidence  clear  that  hard¬ 
ware  and  software  failures  have 
been  properly  partitioned? 

(Note  16) 

_  Have  demonstrated  software 

characteristics  been  highlighted? 
(Note  16) 

_  Have  differences  between  the 

software  tested  and  the  planned 
cperational  software  been 
emphasized?  (Note  17) 

_  Whs  the  test  environmoit  (e.g., 

development,  operational,  or 
maintenance)  appropriate  for  the 
characteristics  to  be 
demonstrated?  (Note  17) 
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Experience  has  shovm  that  unresolved  software  DT&E  issues  are  difficult  to 
resolve  during  CXKE.  Proceeding  to  dedicated  qperational  tests  with  software 
that  has  not  met  its  technical  goals  greatly  increases  the  probability  that 
expensive  and  time-c(»isuming  reworking  will  be  required.  On  a  cost  basis 
alone,  there  may  be  an  order  of  magnitude  difference  betweoi  DT&E  and  OT&E. 
Every  attempt  should  be  made  to  solve  software  problems  before  integration  of 
the  software  and  hardware  (cf .  Note  7) , 

It  should  also  be  recognized  that  a  blurring  of  OT&E  and  DT&E  may  have  takBi 
place.  For  example,  the  suitability  of  the  user  interface  may  have  already 
been  validated  during  ^  early  developmait  test.  Ihe  OT&E  to  date  summary 
^ould  review  the  significance  of  any  such  develcpment  tests  on  cperational 
test  issues.  Included  in  this  discussion  should  be  an  indication  of  whether 
the  operational  test  objectives  have  been  satisfied. 

References  for  Note  19 t  [STEPS,  Blackledge] 


TEMP  Section/Subsection 


Eart  IV  -  OT&E  Outline 

2.  Future  OT&E.  Discuss  all  remain¬ 
ing  OT&E  planned,  beginning  with 
the  date  of  the  current  TEMP 
revision.  Address  SQ)arately 
all  remaining  OT&E,  including 
the  following: 

a.  Ebuionent  Description:  Sum¬ 
mary  of  functional  capability 
and  how  it  is  expected  to 
differ  from  the  production 
model. 

b.  OT&E  (apjectives:  Sunmary  of 
the  specific  OT&E  dDjectives 
to  be  addressed  during  this 
OT&E.  The  objectives  idaiti- 
fied  should  be  the  discrete 
major  goals  of  the  OT&E  effort 
v^ich,  when  achieved  will 
provide  solutions  to  critical 
operational  issues. 

c.  OT&E  Events/lteope  of  Testing/ 
Basic  Scenarios:  Key  OT&E 
events  planned  to  address 
the  cAjjectives.  In  addition 
describe  in  sufficient  detail 
the  scope  of  testing  and  basic 
test  scenarios  so  that  the 
relatiaiship  between  the  test¬ 
ing  and  the  objectives,  and 
the  amount  and  thoroughness  of 
testing  are  clearly  apparent. 

Discuss  the  degree  to  which  the 
test  environment,  including 
procedures  and  threat  simulatiois, 
is  representative  of  the  es^ected 
t^erational  environment.  Discuss 
the  RAM  testing  concept  and  the 
training  and  background  of  the 
operational  test  personnel. 

3.  .CLLtifial  .  All  items 

the  availability  of  \diich  are 
critical  to  the  conduct  of 
adequate  OT&E  prior  to  the  n^t 
decisicsi  point. 


Checklist  Qjestions 


_  Have  differences  between  the 

software  to  be  tested  and  the 
planned  operational  software  been 
sunnarized?  (Note  17) 


_  Are  software  OT&E  test  <±>jectives 

traceable  to  required  software 
characteristics  and  critical  T&E 
issues?  (Note  7,  Note  8) 


Will  the  planned  testing 
demonstrate  the  required  software 
characteristics?  (Note  18) 


_  Do  the  operaticxial  test  analysts 

include  software-trained 
personnel?  (Note  20) 

_  Does  the  RAM  testing  concept 

address  operational  software  T&E 
issues?  (Note  20) 


_  Are  any  new  software  subsystems 

needed  for  OT&E  prior  to  the  next 
decisicn  point?  (Note  13) 

_  Are  any  new  software  support 

systems  or  tools  needed  for  OT&E 
prior  to  the  next  decision  point? 
(Note  14) 
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It  is  ess^tial  that  some  OTStE  personnel  have  software  es^ertise  when  the 
i^stem  contains  critical  software  conponaits.  It  is  also  essential  to 
include  software  in  the  formal  procedures  for  assigning  causes  to  qperational 
eveits. 

Referaices  for  Note  20;  [STEPS ,  Part  2] 
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PAT&E  usually  presents  few  software  issues.  Notable  exceptic^s  to  this  rule 
are  those  instances  in  which  software  plays  a  critical  role  in  manufacturing 
or  production  processes.  In  these  cases,  the  relevant  software  may  be 
treated  as  though  it  were  a  critical  system  conpcMient. 


Uiis  secticxi  provides  a  brief  suintnary 
of  the  resources  for  DTEiE,  OT&E, 
and  PAT&E  that  are  unique  to  the 
program. 

!•  Sg.§fc„i^ttic3.eg .  Identity  the  _  Are  critical  software  components 

actual  number  of  articles/  and  key  subsystems  identified? 

including  key  support  equipment,  (Note  13) 

of  the  system,  required  for  testing 

in  each  phase  and  for  each  major 

type  of  T&E.  If  key  subsystems 

are  to  be  tested  individually 

identify  each  subsystem  and  the 

quantity  required.  ^>ecifically 

idaitify  prototypes,  pilot 

producticxi,  and  producticxi 

models. 

2.  Special  Support  Requirements.  The  _  Is  there  an  explanation  of  how 

special  support  resources  required  test  tools  support  software  test 

for  T&E  with  a  brief  description  objectives?  (Note  22) 

of  the  stqps  being  taken  to 

acquire  them.  _  Are  adequate  steps  being  taken  to 

acquire  each  tool?  (Note  23) 

_  Do  any  of  the  software  testing 

tools  increase  risk?  (Note  24) 
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Most  of  the  effective  software  testing  techniques  (cf.  Note  18)  require 
automated  support  in  the  form  of  testing  tools.  Seme  tools  such  as  test 
drivers  and  file  comparators  are  generic  and  can  be  used  to  support  a  variety 
of  techniques,  tti  the  other  hand,  many  tools  are  specifically  designed  to 
sipport  a  particular  test  methodology*  Hie  tools  that  have  been  chosen 
diould  be  aH>ropriate  to  carry  out  the  planned  tests.  The  lack  of  identified 
test  tools  is  an  indication  ^at  planned  testing  may  be  manueil  and  therefore 
more  labor  intensive  and  error -prone. 

References  for  Note  22 t  [arEP2,  Part  3] 


Note  23t 

Softvare  testing  tools  are  in  short  supply.  Many  of  the  most  successful 
tools  are  proprietary  and  must  be  acquired  from  private  vendors.  Several 
other  tools  have  beai  develcped  in  DoD  labs  and  not  widely  publicized.  As  a 
last  resort,  the  contractor,  project  office  or  test  organization  may  develcp 
a  new  tool  to  sipport  a  specific  software  test, 

Referoice  for  Note  23t  [STEP2,  Fart  3] 


Note  24i 

As  described  in  Note  6,  it  is  possible  that  the  technology  used  in  a  testing 
tool  actually  jnereases  software  risk.  This  is  especially  true  when  a  tool 
must  be  developed  to  support  a  test.  In  that  case,  all  of  the  problems  of 
software  develcproent  can  occur  in  the  acquisition  of  the  new  tool. 

Other  sources  of  risk  are  the  technical  risks  associated  with  undemonstrated 
technologies  and  the  schedule/budget  risks  that  result  from  the  tools  that 
support  some  very  sensitive  test  techniques.  Another  source  of  risk  is  the 
adaptation  of  a  tool  from  one  environmeit  or  project  to  another. 

References  for  Note  24-.  [Bunyard] 
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Mtoinatic  Data  Processing  (ADP) 

to  its  most  general  usage,  this  refers  to  any  use  of  computers  to  process 
information.  Usually,  however,  automatic  data  processing  is  cwit rested 
with  M(XR  applications;  for  exairple,  personnel  and  payroll  applications 
are  ADP  while  weapon  systems  are  MCCR  applications. 

Conpile 

The  process  of  translating  a  computer  program  from  one  programming 
language  (the  source  language)  to  another  (the  object  language) . 

Complete  Functional  Test 

The  process  of  demaistrating  that  each  functional  requirement  is 
satisfied  by  the  software. 

Critical  Software  Conponent 

Any  portion  of  a  computer  program,  any  computer  program  or  any  collection 
of  computer  programs  that  fulfills  a  requirOTait  for  a  Function  as 
described  in  the  TEMP. 

Critical  Software  T&E  Issue 

Any  ai^>ect  of  the  software  system's  capability  that  must  be  questioned 
before  a  system's  overall  worth  can  be  estimated. 

Data  Base  Manager 

Software  that  is  used  to  control  and  access  large  amounts  of  data  that 
are  organized  into  data  bases. 

Data  Item  Description  (DID) 

A  document  that  contains  the  format  and  conteit  preparation  instructions 
for  a  ccptract  deliverable  consisting  of  data  generated  under  work  tasks 
described  within  military  standards. 

Bnulator 

A  program  or  device  that  simulates  the  execution  of  one  computer  by 
another  one. 
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As  applied  to  software,  an  environment  is  the  collection  of 
methodologies,  tools,  machines  and  managenoit  practices  by  which  software 
is  engineered.  Sometimes  the  environment  is  entxxiied  in  an  ext^sive 
piece  of  software;  more  frequently,  the  environmait  is  a  combination  of 
manual  and  automated  procedures  and  methodologies.  Environments  may  be 
distinguished  by  lifecycle  phase  (e.g.,  development  eivironment  or  PDSS 
eivironment) . 

Evolutionary  Acquisition 

The  structuring  of  the  acquisition  process  to  accommodate  evolutionary  or 
incremental  development. 


Goal 

Hie  desired  or  ei^ected  value  of  a  required  software  characteristic. 
Government  Furnished  Equipment  (GFE) 

As  applied  to  software,  C$’E  refers  to  any  software  tool  or  system  that  is 
supplied  by  the  Government  to  a  contractor. 

High  0r<3er  language  (POL) 

Any  programming  language  that  removes  machine  dependencies  or  permits  the 
expression  of  programming  constructs  in  more  natural  terms  than  would 
otherwise  be  possible.  Coinnon  high  order  languages  include  Fortran, 
Jovial,  Ada,  C,  Pascal,  and  CMS-2. 

Host  Computer 

Hie  conputer  on  vAiich  the  software  is  being  developed  or  tested. 

Hie  process  of  designing,  implanaiting,  testing  and  delivering  a  software  ■ 
product  in  increasingly  complete  increments. 

Lifecycle 

Hie  structuring  of  the  phases  and  activities  of  the  design, 
iirplaneitation,  and  operation  process  as  a  function  of  time.  No  single 
lifecycle  adequately  describes  all  software  products.  Model  lifecycles 
are  contained  in  appropriate  Service  standards  and  the  design  documents 
and  standards  of  many  software  developers. 
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Maintenance 


As  applied  to  softvrare/  inaintaiance  refers  to  the  process  of  correcting 
errors,  modifying  deisigns,  and  adding  new  capabilities.  To  avoid 
confusion  between  corrective  maintenance  (correcting  errors)  and  the 
re-engineering  of  software  products,  the  entire  activity  is  sometimes 
referred  to  as  post-develc^ent  or  post-deployment  software  simport 
(PDSS) . 


Mature  Software 

Software  which  has  demonstrated  through  extensive  usage,  testing  and 
repair  of  defects  that  it  meets  design  and  user  requirements.  An 
indication  of  software  maturity  is  the  extent  to  vrtiich  it  is  modified 
after  new  tests. 


Mean  Time  Between  CPerational  Mission  Failure  ( 


PL 


An  operaticml  mission  failure  is  any  system  condition  observed  during 
system  operation  under  c^rational  conditions  that  results  in  a  failure 
to  meet  one  or  more  system  mission  objectives.  Iliese  failures  may  be  due 
to  hardware,  software,  or  operator  failures. 

Mission  Critical  Computer  Resources  (MCX30 


See  Note  1. 

N 

Mission  Critical  Software 

Software  that  implemoits  M(3CR  functions  that  are  essential  to  the 
performance  of  the  system  or  mission  objectives. 

Module  Testing 

Often  used  interchangeably  with  unit  testing.  More  often,  module  tests 
refer  to  tests  of  independently  compiled  software  routines  against 
technical  specifications. 

Post  Deployment  Software  Support  or  Post  Development  Software  Support  (PDSS) 
See  Maintenance. 

Preplanned  Product  Improvement  (P3I) 


An  acguisiticx)  and  design  strategy  that  involves  the  scheduled  and 
planned  enhancanent  of  a  system  or  product. 

Proprietary  Software 

Software  owned  by  an  individual  or  organization  that  has  placed 
restrictions  on  the  use  or  disposition  of  the  software  by  others  who  do 
not  own  it.  lliese  restrictions  are  usually  imposed  by  ^e  commercial 
sector  through  licenses  or  other  agreenoits  that  detail  the  rights  to 
vdiich  the  licoisee  is  entitled. 
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Prototyping 


Qhe  process  of  producing  an  experimoital  versiai  of  a  software  system  or 
portion  of  a  software  i^stem  in  order  to  evaluate  one  or  more  aspects  of 
the  design. 

Randan  Test 

Hie  process  of  supplying  software  input  values  chosen  at  random 
sanpling  from  a  fixed  distribution. 

Beguired  Software  Characteristics 

Parameters  that  are  the  primary  indicators  of  the  software's  conformance 
to  writtai  specificatiais  (the  technical  characteristics)  and  operational 
suitability  and  effectiveness  (operational  characteristics) .  Ideally, 
these  characteristics  should  be  quantitatively  specified  by  the  range  of 
minimally  acceptable  (threshold)  and  desired  (goal)  values  of  the 
parameter. 

Reusable  Software 

Existing  software  that  can  be  inserted  into  use  on  a  given  system  with 
little  or  no  modification. 

Softer?  Engwiggring 

The  practice  of  designing  and  constructing  software  products  using 
disciplined,  caitrolled,  and  monitored  engineering  techniques. 

Sfiftware  Int.gisive 

See  Note  2. 

Softv^re  Quality 

Ihe  extent  to  vAiich  the  softvrare  meets  technical  specifications,  and  user 
needs  and  expectations.  The  totality  of  features  and  characteristics  of 
the  software  that  bear  on  its  ability  to  satisfy  givoi  needs. 

&>.ftKar.g..  Beauirments 

The  statemoits  of  software  systems  c^)ability  that  are  the  basis  of 
software  design.  Ilie  mechanisms  for  describing  softvrare  requironents 
vary  among  the  Services.  For  details  see  MIL-SrD-490  (Specif icatiai 
Practices) ,  DoD-SrD-1679  (Weapon  System  Software  Development) ,  and 
DoD“Sn>*2167  (Defense  fystem  Software  Developmait) . 

Software  Tool 

A  conputer  program  that  provides  automated  support  for  the  development  of 
other  software  products,  fypical  tools  include  conpilers,  editors, 
debuggers,  testing  tools,  librarians,  mail  facilities,  and  various  design 
aids. 
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statement  Coverage 


The  nurrber  or  percentage  of  program  staterooits  that  have  been  executed 
during  testing.  The  utility  of  100%  statement  coverage  is  that  there  may 
be  latent  defects  in  coding  lines  that  have  not  been  executed  during  a 
test. 

Stress  Testing 

The  execution  of  tests  that  attain  or  ^ceed  maxima  defined  by  one  or 
more  software  requirements. 

Systematic  Software  Test 

Any  software  test  that  involves  the  usage  of  a  scieitifically  sound  test 
approach.  Systematic  approaches  should  be  ccxitrasted  with  ad  hoc  tests 
that  may  specify  procedures  and  activities  only. 

Target  Computer 

Bie  operational  conputer. 

Test  Drivers 

Software  that  is  used  to  initiate  or  control  the  executicn  of  the 
software  componaits  being  tested.  The  most  connoi  exanples  of  test 
drivers  occur  during  unit  and  module  tests  wh^  software  subsystems  are 
controlled  by  gaieric  test  drivers  that  simulate  sub^stem  calling 
sequQices  and  provide  for  data  transfer  in  and  out  of  the  subsystem. 

lefft-Hamfiss 

Test  Driver. 

ab££^ld 

The  minimally  acceptable  value  of  a  required  software  characteristic. 

Chit  Testing 

Testing  of  small ,  logically  coheroit  pieces  of  software  (such  as 
subroutines)  against  technical  specif icaticxis. 

User  Interface 

The  (usually  electronic)  interface  betwe^  the  human  and  the  software. 
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